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SUMMARY  PAGE 


Objective:  The  Department  of  the  Navy  (DON)  began  a  Human  Capital  Strategy  (HCS)  agenda  in  2004  and 
incorporated  this  agenda  into  its  objectives  for  2005.  The  HCS  agenda  advances  several  pursuits,  including 
attempts  to  (1)  decrease  overall  manpower  costs  within  the  DON  while  improving  operational  effectiveness, 

(2)  implement  an  integrated  organizational  approach  to  career  progression/advancement,  and  (3)  establish 
actionable  metrics  to  accomplish  these  goals.  One  way  the  Submarine  Force  (SUBFOR)  could  meet  the  HCS 
agenda  would  be  to  select  fewer  officers  for  submarine  training,  which  would  reduce  manpower  costs. 
However,  in  order  to  avoid  a  subsequent  manpower  shortage  from  this  force  reduction  and  thus  maintain 
operational  effectiveness,  SUBFOR  would  need  to  have  an  actionable  metric,  or  predictor,  that  would  allow 
for  the  selection  of  those  officers  for  submarine  training  who  are  the  most  likely  to  remain  and  advance  within 
SUBFOR.  This  study  investigated  whether  or  not  the  SUBSCREEN  test  could  be  used  to  provide  a  predictor 
of  advancement  among  officers  within  SUBEOR.  A  prediction  equation  within  the  SUBSCREEN  test  which 
identifies  the  officers  that  are  the  most  likely  to  remain  within  SUBEOR  could  eventually  provide  the  Navy 
with  an  actionable  metric  that  could  be  used  to  (1)  select  fewer  officers  for  submarine  training,  (2)  reduce 
training  costs,  and  yet  (3)  increase  the  retention  of  the  submarine  officers  who  are  the  most  likely  to  advance  in 
their  careers  and  thus  (4)  maintain  operational  effectiveness. 

Method:  During  attendance  of  Submarine  Officer  Basic  Course  (SOBC),  students  are  required  to  take  the 
SUBSCREEN  test  per  Naval  Submarine  School  (NAVSUBSCOE)  Instruction  6420.1  (see  Appendix  A). 
SUBSCREEN  is  a  240-item  test  of  mental  health  functioning,  motivation,  and  environmental  adaptability.  The 
SUBSCREEN  test  is  used  to  evaluate  the  “psychological  fitness”  of  Naval  personnel  for  submarine  training 
and  duty  as  mandated  by  the  Manual  of  the  Medical  Department  of  the  U.S.  Navy  (MANMED)  in  Article  15- 
69,  paragraph  2(i)  (see  Appendix  B).  Consequently,  this  study  was  undertaken  to  determine  if  the 
SUBSCREEN  test  could  provide  an  actionable  metric  which  predicts  which  of  the  SOBC  students  are  the  most 
likely  to  progress  in  their  careers  to  the  Submarine  Officer  Advanced  Course  (SOAC),  and  thus  advance  to 
Department  Head  billets  within  the  Submarine  Eleet.  To  wit,  a  database  of  submarine  officers  who  had 
ascended  to  SOAC  was  combined  with  a  database  of  submarine  officers  who  were  not  selected  for  SOAC,  and 
these  two  known-groups  were  compared  via  SUB  SCREEN  test  scores  to  identify,  retrospectively,  scales  which 
ultimately  predicted  advancement  to  SOAC. 

Findings:  Using  correlation  and  regression  techniques,  we  found  that  six  motivational  scales  and  one 
response-set  scale  within  the  SUBSCREEN  test  formed  an  optimal  linear  composite  that  predicted  which 
SOBC  students  were  likely  to  advance  to  SOAC,  with  R  =  .25  (p  <  .001).  Several  advantageous  initial 
outpoints  were  identified  for  this  prediction  equation,  which  was  termed  the  Submarine  Officer  Retention  Test, 
or  SORT. 

Application:  SORT,  and  the  motivational  traits  that  comprise  the  SORT,  could  be  used  as  actionable  metrics 
that  enhance  officer  selection  decisions  for  submarine  training.  However,  further  tests  of  this  application  of 
SUBSCREEN’s  SORT  may  be  required.  If  administration  of  the  SUBSCREEN  test  were  to  be  placed  earlier 
in  the  accession  pipeline,  e.g.,  prior  to  Nuclear  Power  School  rather  than  during  SOBC,  this  administrative 
change  could  moderate  the  accuracy  of  SORT’S  predictions,  and  this  possibility  would  require  empirical 
investigation. 


ADMINISTRATIVE  INFORMATION 

This  investigation  was  conducted  under  Work  Unit  #50501  (formerly  #5403)  entitled  “SUBSCREEN,”  which 
is  funded  by  CNO-N779  via  NAVSEA  (N0002404WX01546).  Eunding  was  also  provided  by 
COMSUBNAVEOR.  The  views  expressed  in  this  report  are  those  of  the  authors  and  do  not  reflect  the  official 
policy  or  position  of  the  Department  of  the  Navy,  Department  of  Defense,  or  the  U.S.  Government.  This  report 
was  approved  for  publication  on  28  March  2005  and  has  been  designated  as  NSMRE  Technical  Report  TR 
#1238. 
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ABSTRACT 


The  Department  of  the  Navy  (DON)  began  a  Human  Capital  Strategy  (HCS)  agenda  in  2004  and 
incorporated  this  agenda  into  its  objectives  for  2005.  The  HCS  agenda  advances  several  pursuits, 
including  attempts  to  (1)  decrease  overall  manpower  costs  within  the  DON  while  improving 
effectiveness,  (2)  implement  an  integrated  organizational  approach  to  career 
progression/advancement,  and  (3)  establish  actionable  metrics  to  accomplish  these  goals.  One 
way  the  Submarine  Force  (SUBFOR)  could  meet  the  HCS  agenda  would  be  to  select  fewer 
officers  for  submarine  training,  which  would  reduce  manpower  costs.  However,  in  order  to  avoid 
a  subsequent  manpower  shortage  from  this  force  reduction  and  thus  maintain  operational 
effectiveness,  SUBFOR  would  need  to  have  an  actionable  metric,  or  predictor,  that  would  allow 
for  the  selection  of  those  officers  for  submarine  training  who  are  the  most  likely  to  remain  and 
advance  within  SUBFOR.  This  study  investigated  whether  or  not  the  SUBSCREEN  test  could  be 
used  to  provide  a  predictor  of  advancement  among  officers  within  SUBEOR.  SUBSCREEN  is  a 
240-item  test  of  mental  health  functioning,  motivation,  and  environmental  adaptability  used  to 
evaluate  the  “psychological  fitness”  of  Naval  personnel  for  submarine  training  and  duty.  Using 
correlation  and  regression  techniques,  we  found  that  six  motivational  scales  and  one  response-set 
scale  within  the  SUBSCREEN  test  formed  an  optimal  linear  composite  that  predicted  which 
Submarine  Officer  Basic  Course  (SOBC)  students  were  likely  to  advance,  approximately  seven 
years  later,  to  the  Submarine  Officer  Advanced  Course  (SOAC),  with  R  =  .25  (p<  .001).  Several 
advantageous  initial  outpoints  were  identified  for  this  prediction  equation,  which  was  termed  the 
Submarine  Officer  Retention  Test,  or  SORT.  In  the  future,  the  SORT,  and  the  motivational  traits 
that  comprise  the  SORT,  could  be  used  as  actionable  metrics  that  enhance  officer  selection 
decisions  for  submarine  training.  However,  further  tests  of  this  application  of  SUBSCREEN’ s 
SORT  may  be  required  as  administration  of  the  SUBSCREEN  test  could  profitably  be  placed 
earlier  in  the  training  pipeline,  prior  to  Nuclear  Power  School  rather  than  during  SOBC,  and  this 
administrative  change  could  moderate  the  accuracy  of  SORT’S  predictions. 
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INTRODUCTION 


Trying  to  do  more  with  less  is  an  ongoing  struggle  in  the  Department  of  Defense,  as  budgets 
remain  relatively  stable  while  demands  increase.  In  2004  the  Department  of  the  Navy  (DON) 
began  a  Human  Capital  Strategy  (HCS)  agenda,  and  incorporated  this  agenda  into  its  objectives 
for  2005.  The  HCS  agenda  advances  several  pursuits,  including  attempts  to  (1)  decrease  overall 
manpower  costs  within  the  DON  while  improving  effectiveness,  (2)  implement  an  integrated 
organizational  approach  to  career  progression/advancement,  and  (3)  establish  actionable  metrics 
to  accomplish  these  goals.  The  HCS  agenda  is  Navy-wide,  and  all  forces,  including  the 
Submarine  Force  (SUBFOR),  are  attempting  to  pursue  HCS  goals.  One  way  SUBFOR  could 
meet  part  of  the  HCS  agenda  would  be  to  select  fewer  officers  for  submarine  training,  which 
would  reduce  manpower  costs.  However,  in  order  to  avoid  a  subsequent  manpower  shortage 
from  this  force  reduction  and  thus  maintain  operational  effectiveness,  SUBFOR  would  need  to 
have  an  actionable  metric,  or  predictor,  that  would  allow  for  the  selection  of  those  officers  for 
submarine  training  who  are  the  most  likely  to  remain  and  advance  within  SUBFOR. 

Submarine  Officer  Basic  Course  (SOBC)  is  the  first  sub-specific  step  in  training  sub  officers, 
and  is  only  offered  at  SUBASE  NEON.  During  attendance  of  SOBC,  students  are  required  to 
take  the  SUBSCREEN  test  per  Naval  Submarine  School  (NAVSUBSCOE)  Instruction  6420.1 
(see  Appendix  A).  SUBSCREEN  is  a  240-item  test  of  mental  health  functioning,  motivation,  and 
environmental  adaptability  (Bing  &  Eisenberg,  2004).  The  SUBSCREEN  test  is  used  to  evaluate 
the  “psychological  fitness”  of  Naval  personnel  for  submarine  training  and  duty  as  mandated  by 
the  Manual  of  the  Medical  Department  of  the  U.S.  Navy  (MANMED)  in  Article  15-69, 
paragraph  2(i)  (see  Appendix  B).  SUBSCREEN  is  administered  and  maintained  by  Naval 
Submarine  Medical  Research  Eaboratory  (NSMRE),  and  used  by  NAVSUBSCOE  to  screen  both 
SOBC  and  Basic  Enlisted  Submarine  School  (BESS)  students  for  potential  incompatibilities 
(e.g.,  claustrophobia)  with  the  submarine  environment.  SUBSCREEN  consists  of  many  scales 
that  measure  Affective  (e.g.,  situational  control,  depression,  nervousness).  Socialization  (e.g., 
aggression,  social  isolation,  impulsiveness,  coercive  tendencies),  and  Motivational  (e.g.,  mistake 
joining  submarines,  uncertain  about  submarines)  factors.  SUBSCREEN  also  contains  scales  that 
detect  response  distortion  (e.g.,  faking  good),  and  the  scoring  program  uses  additional  algorithms 
to  identify  response  patterns  indicative  of  faking  attempts.  Additional  scales  are  composed  of 
items  used  to  assess  suicidal  ideation,  claustrophobia,  self-criticism,  unusual  thoughts,  and 
unusual  physical  complaints.  Students  respond  to  each  item  on  the  SUB  SCREEN  test  by  using  a 
five-point  Eikert  scale  with  the  following  response  options:  agree  strongly,  agree,  does  not  apply, 
disagree,  and  disagree  strongly. 

Currently,  each  SOBC  student’s  scale  scores  are  compared  to  the  scale  score  norms  (i.e.,  means 
and  standard  deviations)  obtained  from  over  4,000  SOBC  students  who  have  taken  the  test  over 
the  last  10  years.  Approximately  12%  of  SOBC  students  are  referred  to  the  Mental  Health  Clinic 
of  the  Naval  Ambulatory  Care  Center  (NACC)  in  Groton  for  a  mental  health  evaluation  because 
of  their  SUBSCREEN  test  scores.  In  general,  for  any  scale  score  to  warrant  a  referral,  it  must  be 
more  extreme  (i.e.,  higher  or  lower  depending  upon  the  referral  criterion)  than  approximately 
98%  of  the  other  scores  in  the  SOBC  student  norms  database.  In  other  words,  the  score  must 
land  in  or  above  the  98‘^  percentile  (i.e.,  -1-2.054  standard  deviations  above  the  mean)  to  warrant 
referral,  or  possibly  on  or  below  the  2"‘*  percentile  (i.e.,  -2.054  standard  deviations  below  the 
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mean)  to  warrant  referral,  depending  upon  the  referral  criterion.  There  are  over  20  standardized 
referral  criteria  in  the  scoring  program  for  the  test,  and  thus  over  20  possible  reasons  for  referral 
and  further  evaluation  within  any  single  SUBSCREEN  test  profile.  As  many  SUBSCREEN 
scales  are  intercorrelated,  the  SUBSCREEN  profiles  for  students  requiring  further  evaluation 
often  contain  more  than  one  referral  reason. 

As  stated  above,  the  SUBSCREEN  test  is  administered  by  NSMRE,  and  the  SUBSCREEN 
answer  sheets  are  subsequently  scored  at  NSMRE.  SOBC  students  who  have  indications  or  an 
indication  of  incompatibility  with  submarine  training  and/or  duty  (e.g.,  are  +1.96  standard 
deviations  above  the  mean  on  the  suicidal  ideation  scale)  according  to  their  SUB  SCREEN 
profile  are  referred  to  the  Mental  Health  Clinic  for  a  mental  health  status  interview  and  further 
evaluation.  The  Mental  Health  Clinic  receives  a  list  of  names  of  those  to  be  interviewed  as  well 
as  the  SUBSCREEN  profile  report,  which  contains  the  SUBSCREEN  test  scores  and  the  specific 
referral  reasons.  The  SUB  SCREEN  profile  reports  and  the  referral  reasons  aid  the  interviews  and 
help  to  guide  the  subsequent  final  dispositions  reached  by  the  Mental  Health  Clinic. 
NAVSUBSCOE  receives  only  the  list  of  names  of  those  to  be  referred  to  the  Mental  Health 
Clinic  in  order  to  coordinate  the  interviews  of  their  students  with  other  SOBC  activities  (e.g., 
classes,  meetings,  damage  control  training  sessions,  etc.).  After  the  conclusion  of  the  mental 
health  status  interview  and  perhaps  further  testing  and/or  treatment,  the  Mental  Health  Clinic 
reaches  one  of  three  recommended  final  dispositions  for  the  student:  (1)  qualified  for  submarine 
training  and  duty  and  recommend  a  return  to  SOBC  for  training,  (2)  unqualified  for  submarine 
training  and  duty  and  recommend  a  transfer  to  the  Surface  Elect,  and  (3)  unqualified  for 
submarine  training  and  duty  and  recommend  administrative  separation  (ADSEP).  The  final 
disposition  is  sent  to  NAVSUBSCOE  as  a  recommendation.  The  leadership  at  NAVSUBSCOE 
then  decides  the  final  outcome  for  the  SOBC  student.  In  sum,  the  SUBSCREEN  test  occupies  the 
first  step  of  a  mandatory,  multi-step  “select  out”  process  used  to  evaluate  the  “psychological 
fitness”  of  Naval  personnel  for  submarine  training  and  duty  (MANMED,  Article  15-69, 
paragraph  2(i)). 

A  recent  innovation  in  the  SUBSCREEN  program  came  in  the  form  of  a  new  actionable  metric 
within  the  SUBSCREEN  test,  which  was  created  in  the  fall  of  2001.  This  actionable  metric,  or 
empirical  predictor,  was  based  on  SUBSCREEN  scale  scores,  and  was  created  for  the  purpose  of 
predicting  fleet  attrition  among  BESS  students  (Bing  &  Eisenberg,  2003a).  This  predictor  was 
eventually  named  the  Submarine  Attrition  Risk  Scale,  or  SARS,  and  then  renamed  to  the 
SubMarine  Attrition  Risk  Test,  or  SMART  (Bing  &  Eisenberg,  2003b,  2004). 

The  SMART  is  an  optimal  linear  composite  of  scales  within  the  SUBSCREEN  test  that  predict 
adverse  (e.g.,  misconduct,  substance  abuse)  and  early  (i.e.,  prior  to  end  of  first  enlistment)  fleet 
attrition  among  BESS  students  (Bing  &  Eisenberg,  2004).  The  SMART  does  not  provide  a 
psychological  diagnosis,  but  simply  provides  the  probability  with  which  one  can  expect  a  BESS 
student  to  separate  for  negative  reasons  once  he  reaches  the  fleet.  The  SMART  has  been 
prospectively  validated  against  early  attrition  indicators,  such  as  nonjudicial  punishment  (NJP) 
events  and  legal  hold  status  for  alleged  criminal  activities  (Bing  &  Eisenberg,  2003b). ^  The 


*  “. .  .NJP. .  .refers  to  certain  limited  punishments  which  can  be  awarded  for  minor  disciplinary  offenses  by  a 
commanding  officer  or  officer  in  charge  to  members  of  his/her  command.  In  the  Navy  and  Coast  Guard,  nonjudicial 
punishment  proceedings  are  referred  to  as  ‘captain's  mast’  or  simply  ‘mast’”  (Powers,  2005,  p.  1). 
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SMART  is  a  statistically  significant  predictor  of  these  early  attrition  indicators,  and  the  higher  an 
enlisted  student  scores  on  the  SMART,  the  more  likely  he  is  to  commit  an  act  that  will  eventually 
result  in  disciplinary  action  (e.g.,  NJP)  and  separation  from  the  Navy. 

In  sum,  the  SMART  is  a  subset  of  the  SUBSCREEN  test,  and  potentially  useful  for  evaluation, 
treatment,  and  separation  decisions  made  for  BESS  students.  As  a  result,  the  use  of 
SUBSCREEN’s  SMART  may  eventually  help  the  Navy  to  meet  HCS  objectives  within  the 
enlisted  submarine  community.  In  fact,  currently  a  6.4,  BUMED-funded  Test  and  Evaluation 
(T&E)  of  the  SMART  is  underway  to  determine  whether  or  not  SUBSCREEN’s  SMART  can  be 
used  to  increase  performance  and  retention  among  BESS  students  (Bing,  2005;  see  Appendix 
C).  However,  currently  the  SUBSCREEN  program  does  not  have  an  actionable  metric  that  is 
analogous  to  the  SMART  for  the  officer  submarine  community.  Consequently,  this  study  was 
undertaken  to  determine  if  the  SUBSCREEN  test  could  provide  an  actionable  metric  which 
predicts  which  of  the  SOBC  students  are  the  most  likely  to  advance  in  their  careers  to  the 
Submarine  Officer  Advanced  Course  (SO AC),  and  thus  progress  to  Department  Head  billets 
within  the  Submarine  Eleet.  A  prediction  equation  within  the  SUBSCREEN  test  which  identifies 
the  officers  that  are  the  most  likely  to  remain  within  SUBEOR  could  eventually  provide  the 
Navy  with  an  actionable  metric  that  could  be  used  to  (1)  select  fewer  officers  for  submarine 
training,  (2)  reduce  training  costs,  and  yet  (3)  increase  the  retention  of  the  submarine  officers 
who  are  the  most  likely  to  advance  in  their  careers  and  thus  (4)  maintain  operational 
effectiveness. 


HYPOTHESES 

Only  those  submarine  officers  with  high  to  very  high  levels  of  job  performance  advance  from 
SOBC  to  SOAC.  In  fact,  for  every  four  SOBC  students,  approximately  only  one  (or  25%)  will 
ultimately  attend  SOAC.  As  fluctuations  in  motivation  tend  to  impact  job  performance  (Barrick, 
Stewart,  &  Piotrowski,  2002;  Helmreich,  Sawin,  &  Carsrud,  1986;  Kirk  &  Brown,  2003; 
Mitchell,  1997),  we  hypothesized  that  the  motivational  scales  within  the  SUBSCREEN  test 
would  be  the  best  predictors  of  ascendancy  from  SOBC  to  SOAC.  However,  we  also  examined 
other  SUBSCREEN  scales,  especially  those  that  were  included  in  the  SMART  (see  above)  for 
predicting  attrition  among  enlisted  submariners,  to  determine  if  they  could  also  explain 
ascendancy  from  SOBC  to  SOAC  among  officers. 

METHOD 


Participants 

SOBC  students  who  took  the  SUBSCREEN  test  per  MANMED  requirement  (Article  15-69, 
paragraph  2(i))  between  October  of  1993  and  September  of  1999  served  as  the  participants  in  the 
current  study.  Data  on  selection  into  SOAC  for  these  SOBC  students  was  obtained  from  the 
Nuclear  Officer  Program  Manager  (N133C).  The  time  between  SUBSCREEN  test  administration 
at  SOBC  and  selection  into  SOAC  is  approximately  seven  years.  Consequently,  at  the  time  of 
this  study  (March  of  2005),  selection  into  SOAC  or  rejection  from  SOAC  could  not  be  known 
for  many  of  those  tested  between  October  of  1998  and  September  of  1999  because  these  students 


^  Appendix  C  provides  a  short  discussion  on  the  use  of  psychological  tests  for  employment  decisions. 
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had  not  been  in  the  Navy  a  suffieient  amount  of  time  for  their  SOAC  seleetion  deeisions  to  have 
been  made.  Thus,  these  partieipants  were  ineluded  in  the  database  only  if  they  had  been  seleeted 
into  SOAC,  but  were  eliminated  from  the  database  otherwise  beeause  it  eould  not  be  assumed 
that  they  had,  as  of  yet,  failed  make  SOAC  seleetion.^ 

Predictor  Measure 

The  SUBSCREEN  test  was  used  to  measure  all  of  the  predietors  examined  in  this  study. 
SUBSCREEN  eonsists  of  many  seales  that  measure  Affective  (e.g.,  depression,  nervousness). 
Socialization  (e.g.,  aggression,  coercive  tendencies),  and  Motivational  (e.g.,  mistake  joining 
submarines,  uncertain  about  submarines)  factors.  SUBSCREEN  also  contains  scales  that  detect 
response  distortion  (e.g.,  faking  good),  and  the  scoring  program  uses  additional  algorithms  to 
identify  response  patterns  indicative  of  faking  attempts.  Additional  scales  are  composed  of  items 
used  to  assess  suicidal  ideation,  claustrophobia,  self-criticism,  unusual  thoughts,  and  unusual 
physical  complaints.  Eive  to  14  items  are  used  to  measure  each  disposition  depending  upon  the 
scale.  Students  respond  to  each  item  by  using  a  five-point  Eikert  scale  with  the  following 
response  options:  agree  strongly,  agree,  does  not  apply,  disagree,  and  disagree  strongly. 

Criterion  Measure 

Selection  and  ascendancy  into  SOAC  was  coded  as  1,  and  not  being  selected  for  SOAC  was 
coded  as  0.  This  dichotomous  SOAC  ascendancy  outcome  variable  served  as  the  criterion. 

Procedure 

Eor  this  study  we  utilized  a  known-group  retrospective  approach  to  develop  an  empirical 
predictor  of  advancing  to  SOAC  from  SUBSCREEN  test  scores  obtained  on  SOBC  students. 
Specifically,  a  known-group,  retrospective  approach  was  utilized  to  determine  if  scales  within 
the  SUBSCREEN  test  could  predict  which  SOBC  students  were  the  most  likely  to  advance  in 
their  careers  to  SOAC,  and  thus  progress  to  Department  Heads  within  the  Submarine  Elect.  Thus, 
a  known-group  of  submarine  officers  who  were  selected  for  SOAC  was  compared  to  a  known- 
group  of  submarine  officers  who  did  not  reach  SOAC,  either  because  they  were  not  selected  for 
SOAC,  voluntarily  withdrew  from  SUBEOR  prior  to  the  SOAC  selection  process,  or  disqualified 
and  separated  from  SUBEOR  prior  to  SOAC.  These  two  known-groups,  those  who  advanced  to 
SOAC  and  those  who  did  not  advance,  were  compared  in  terms  of  their  SUBSCREEN  test 
scores,  and  all  of  these  submarine  officers  had  taken  the  SUBSCREEN  test  when  attending 
SOBC.  Thus,  two  databases  of  submarine  officers,  those  who  ascended  to  SOAC  and  those  who 
did  not,  were  merged,  and  correlation  and  regression  analyses  were  conducted.  SUB  SCREEN 
scales  that  were  statistically  significant  in  terms  of  predicting  ascendancy  to  SOAC  in 
hypothesized  directions,  either  when  included  as  part  of  a  multiple  logistic  regression  equation  or 
in  terms  of  the  zero-order  correlation  with  the  dichotomous  outcome  (i.e.,  ascended  to  SOAC,  1, 
versus  did  not  ascend,  0),  were  selected  for  inclusion  into  a  new  composite  scale  that  has  been 
termed  the  Submarine  Officer  Retention  Test,  or  SORT. 


^  Eliminating  all  students  tested  between  October  of  1998  and  September  of  1999  did  not  change  the  study’s 
findings. 
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RESULTS 


In  order  to  maintain  the  security  of  the  SUBSCREEN  test,  the  six  motivational  scales  that  were 
found  to  predict  SO  AC  ascendancy  will  be  referred  to  as  EMI  through  EM5  and  PMl.  EMI 
through  EM5  are  scales  that  assess  a  lack  of  motivation,  and  high  scores  on  these  scales  are 
indicative  of  motivational  problems.  PMl  is  a  scale  that  assesses  a  positive  motivational 
characteristic  for  submarine  service,  and  high  scores  on  this  scale  are  indicative  of  high  levels  of 
job-relevant  motivation.  Eikewise,  the  response-set  scale  also  found  to  predict  SOAC 
ascendancy  will  be  referred  to  as  RS 1 .  High  scores  on  the  RS 1  measure  are  indicative  of  the 
presence  of  dysfunctional  attitudes,  dysfunctional  traits,  and/or  attempts  at  response  distortion. 
The  actual  names  of  these  SUBSCREEN  scales  and  their  respective  items  can  be  viewed  in 
Appendix  D,  which  is  for  official  use  only,  and  can  be  requested  from  the  Commanding  Officer, 
Naval  Submarine  Medical  Research  Eaboratory,  Groton,  CT. 

When  all  seven  predictors  (EMI,  EM2,  EM3,  EM4,  EM5,  PMl,  &  RSI)  were  included  in  a 
logistic  regression  equation  with  advancement  to  SOAC  as  the  criterion,  we  found  R  =  .25,  p  < 
.001.  Thus,  approximately  6.25%  of  the  variance  in  advancement  to  SOAC  was  explained  by  this 
optimal  linear  composite  found  within  the  SUB  SCREEN  test.  Several  advantageous  initial 
outpoints  were  identified  for  this  prediction  equation,  which  was  termed  the  Submarine  Officer 
Retention  Test,  or  SORT.  More  detailed  results  regarding  the  creation  of  the  SORT,  the  results 
for  the  individual  scales  of  which  SORT  is  composed,  and  the  findings  from  the  outpoint 
analyses  are  described  below. 

We  first  correlated  the  dichotomous  outcome,  SOAC  ascendancy,  with  each  SUBSCREEN  scale 
including  SUBSCREEN’ s  SMART.  Of  these  27  zero-order  correlations,  six  were  statistically 
significant.  All  six  of  these  correlations  were  obtained  by  SUBSCREEN’ s  motivational  scales, 
and  thus  our  hypothesis  that  the  motivational  scales  would  be  the  best  predictors  of  ascendancy 
from  SOBC  to  SOAC  was  confirmed.  These  correlations  are  presented  in  Table  1  along  with  the 
internal  consistency  reliability  estimates  for  each  scale."^ 

Table  1 

Correlations  Between  Motivational  Scales  and  SOAC  Ascendancy 


Scale 

SOAC  Ascendancy 

Coefficient  Alpha 

EMI 

-.054* 

.66 

EM2 

-.085** 

.66 

EM3 

-  102** 

.81 

EM4 

-.133** 

.71 

EM5 

-.091** 

.68 

PMl 

_l4g** 

.74 

Note,  n  ranges  from  1,473  to  1,524  because  of  missing  data. 
*  p  <  .05.  **  p  <  .01. 


^  Coefficient  alpha  was  used  to  estimate  internal  consistency  reliability  for  each  scale.  Coefficient  alpha  provides  an 
estimate  of  the  percentage  of  variance  in  scale  scores  that  is  a  function  of  true  variance  in  the  attribute  being 
assessed  rather  than  measurement  error.  One  minus  coefficient  alpha  provides  an  estimate  of  the  percentage  of 
variance  in  scale  scores  due  to  measurement  error.  Internal  consistency  reliability  estimates  of  .70  and  higher  are 
generally  regarded  as  acceptable  levels  for  psychological  tests  (Nunnally,  1978).  Coefficient  alpha  for  RS  1  was  .86. 
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Consequently,  all  of  these  scales  were  selected  for  inclusion  into  a  multiple  logistic  regression 
equation  used  to  predict  SOAC  ascendancy.  We  also  utilized  the  ordinary  least  squares  (OLS) 
multiple  regression  technique  for  comparison  purposes,  but  because  of  the  nature  of  the 
dichotomous  outcome  assumed  that  the  logistic  technique  would  yield  more  accurate  results  as 
OLS  can  result  in  impossible  predicted  outcome  values  (i.e.,  below  0  and  above  1).  Although, 
RSI  (coefficient  alpha  of  .86)  did  not  obtain  a  significant  zero-order  correlation  with  SOAC 
ascendancy  (r  =  -.020,  p  >  .  10),  we  included  it  in  the  regression  analyses  because  it  is  a  predictor 
of  attrition  among  enlisted  submariners  and  thus  an  integral  component  of  the  SMART.  Also,  by 
including  RSI  in  the  regression  analyses  we  were  able  to  determine  if  RSI  could  predict 
variance  in  SOAC  ascendancy  above  and  beyond  that  predicted  by  the  motivational  scales,  and 
once  variance  in  motivation  had  been  partialled  from  the  response-set  assessed  by  RSI.  Table  2 
presents  the  results  from  the  OLS  and  logistic  regression  equations. 

Table  2 

Unstandardized  Regression  Coejficients  for  All  Six  Scales  in  Predicting  SOAC  Ascendancy 


SOAC  Ascendancy 

Scale 

OLS  Regression 

Logistic  Regression 

LMl 

-.040 

-.267 

LM2 

-.042 

-.303 

LM3 

-.024 

-.396 

LM4 

-.064^ 

-.403^ 

LM5 

-.123** 

-.851** 

PMl 

.140** 

.901** 

RSI 

-.005** 

-.039** 

Note,  n  was  1,437  for  these  analyses  due  to  listwise  elimination  of  cases  with  missing  values. 
The  scale  of  measurement  for  RSI  is  much  greater  than  that  for  the  other  scales,  hence  the 
statistical  significance  for  the  unstandardized  regression  coefficient  of  -.005  for  RSI. 

^p<.10.  *p<.05.  **p<.0L 

As  can  be  seen  from  these  analyses,  all  of  the  predictors  continued  to  be  associated  with  SOAC 
ascendancy  in  predicted  directions  (e.g.,  LMl  through  LM5  remained  negatively  associated  with 
SOAC  ascendancy),  even  though  not  all  of  them  remained  statistically  significant.  Additionally, 
RSI  clearly  predicted  a  statistically  significant  portion  of  variance  in  SOAC  ascendancy  in  the 
theoretically  defensible  direction  in  the  presence  of  the  motivational  scales.  Using  hierarchical 
OLS  regression,  we  found  that  RSI  predicted  an  additional  2.3%  of  the  variance  in  SOAC 
ascendancy  above  and  beyond  that  predicted  by  the  motivational  scales  (AR^  =  .023,  p  <  .01). 

Using  the  prediction  equations  obtained  from  the  OLS  and  logistic  analyses,  we  calculated 
predicted  values  of  SOAC  ascendancy  for  each  participant,  and  correlated  these  values  with  the 
actual  SOAC  ascendancy  dichotomous  outcome.  These  results  are  presented  in  Table  3. 
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Table  3 


Correlations  Between  OLS  and  Logistic  Predictions  and  SOAC  Ascendancy 


Variable 

1 

2 

3 

Eogistic  Predicted  Values  “ 

OES  Predicted  Values 

SOAC  Ascendancy 

.955** 

.250** 

.229** 

Note,  n  was  1,437  for  these  analyses  due  to  listwise  elimination  of  cases  with  missing  values. 

^  Predicted  values  from  the  logistic  regression  equation  were  ultimately  termed  Submarine 
Officer  Retention  Test  scores,  or  SORT  scores. 

**  p  <  .01. 

As  can  be  seen  from  these  analyses,  the  OLS  and  logistic  prediction  equations  yielded  a  nearly 
identical  rank  order  of  predicted  outcomes,  with  r  =  .955  ip  <  .01)  between  these  predicted 
values.  However,  the  predicted  values  from  the  logistic  regression  equation  correlated  slightly 
higher  with  the  actual  SOAC  ascendancy  outcome  in  comparison  to  the  OLS  equation,  with  R  = 
.250  (p  <  .001)  for  logistic  regression  andi?  =  .229  (p  <  .001)  for  OLS  regression,  respectively. 
Consequently,  the  logistic  regression  equation  was  chosen  as  the  preferred  prediction  equation. 

Probability  of  =  exp((-.267*LMl)+(-.303*LM2)+(-.396*LM3)+(-.403*LM4)+ 

Ascending  to  (-.851*LM5)+(.901*PMl)+(-.039*RSl)+L210)  / 

SOAC  (SORT)  (l+exp((-.267*LMl)+(-.303*LM2)+(-.396*LM3)+(-.403*LM4)+ 

(-.851*LM5)+(.901*PMl)+(-.039*RSl)+L210)). 

Examination  of  this  logistic  regression  equation  reveals  the  following:  (1)  As  scale  scores  for 
LMl  through  LM5  increase,  the  probability  of  ascending  to  SOAC  decreases,  (2)  As  scale  scores 
for  PMl  increase,  the  probability  of  ascending  to  SOAC  increases,  and  (3)  As  scale  scores  for 
RSI  increase,  the  probability  of  ascending  to  SOAC  decreases.  As  each  regression  coefficient  for 
these  scales  remained  in  the  predicted  direction  (see  Table  1),  each  of  these  scales  was  retained 
for  the  final  regression  equation  even  though,  as  stated  above,  not  all  of  the  respective  regression 
coefficients  were  statistically  significant  (see  Table  2). 

Were  these  scales  to  belong  to  different  tests  that  are  time-consuming  and  costly  to  administer,  it 
would  be  advisable  to  select  only  those  scales  that  retained  statistically  significant  regression 
coefficients  for  the  ultimate  creation  of  the  actionable  metric,  or  prediction  equation.  However, 
in  this  case  each  scale  is  contained  within  the  240-item  SUBSCREEN  test,  so  reducing  the 
number  of  scale  predictors  to  save  time  and  money  is  not  a  factor.  Also,  it  is  often  noted  that 
redundancy  in  psychological  measurement  is  advantageous  for  obtaining  more  reliable,  and 
ultimately  more  valid,  assessments  of  individual  differences.  Indeed,  the  psychometric  reliability 
of  a  test  is  partly  a  function  of  necessary  redundancy  in  the  test’s  item  content  (Nunnally  & 
Bernstein,  1994).  Thus,  redundancy  of  scales  for  predicting  important  outcomes,  such  as  SOAC 
ascendancy,  is  also  advantageous  because  it  ultimately  results  in  a  more  reliable  and  valid 
prediction  equation.  Also,  when  feasible,  several  measures  (e.g.,  scales,  tests,  etc.)  should  be 
utilized  when  generating  a  product  that  may  be  used  to  enhance  important  human  resource 
decisions  (e.g.,  selection  for  training  opportunities)  as  these  decisions  should  not  be  based  on  just 
one  factor.  Thus,  we  chose  to  maintain  the  redundancy  of  these  scales  as  predictors  in  the  final 
regression  equation  because  (1)  their  redundancy  should  increase  the  validity  of  the  ultimate 
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prediction  as  noted,  (2)  their  zero-order  correlations  with  SOAC  ascendancy  were  statistically 
significant,  (3)  these  correlations  were  in  the  predicted  directions,  and  (4)  the  corresponding 
regression  coefficients  remained  in  the  predicted  directions  when  all  scales  were  entered  into  the 
prediction  equation. 

In  sum,  using  the  logistic  regression  technique  we  found  that  these  six  SUBSCREEN  scales 
could  be  used  to  form  an  optimal  linear  composite  that  predicted  advancement  to  SOAC.  We 
termed  this  prediction  equation  the  Submarine  Officer  Retention  Test,  or  SORT.  Table  4  presents 
the  mean  and  standard  deviation  for  the  predicted  values  from  the  SORT  along  with  the  mean 
and  standard  deviation  for  the  dichotomous  outcome  variable  of  SOAC  ascendancy. 

Table  4 


Minimums,  Maximums,  Means  and  Standard  Deviations  for  SORT  and  SOAC  Ascendancy 


SORT 

SOAC  Ascendancy 

Min 

Max 

M 

SD 

Min 

Max 

M 

SD 

.000 

.630 

.223 

.100 

.000 

1.000 

.226 

.419 

Note,  n  was  1,437  for  SORT  and  1534  for  SOAC  ascendancy. 


Clearly,  the  mean  value  obtained  from  the  SORT  equation  (M  =  .223)  is  very  close  to  the  base 
rate  of  advancement  to  SOAC  in  the  current  sample,  which  is  approximately  23%  and  given  by 
the  mean  of  the  dichotomous  SOAC  ascendancy  outcome  variable.  Eigure  1  displays  a  histogram 
of  SORT  scores  for  the  two  known-groups,  those  who  were  not  selected  for  SOAC  versus  those 
who  attended  SOAC. 

Figure  1 

Histogram  of  SORT  Scores  Displayed  by  SOAC  Ascendancy  Groups 


Count 


SORTT  Scores 


SOAC  Ascendancy 

□  Not  Selected 

□  Attended  SOAC 
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As  can  be  seen  from  Figure  1,  the  SORT  is  capable  of  diseriminating  between  the  two  groups, 
and  various  outpoints  can  be  selected  to  investigate  the  discriminatory  power  of  this  prediction 
equation.  Thus,  several  outpoints  were  identified  for  the  SORT,  and  the  results  from  these 
eutpoint  analyses  are  presented  in  Table  5. 

Table  5 

SORT  Category  by  SO  AC  Ascendancy  Crosstabulation 


SOAC  Ascendancy 

Total 

No  (0) 

Yes  (1) 

SORT  Category 

Low 

Count 

195 

18 

213 

%  within  SORT  Category 

91.5% 

8.5% 

100.0% 

%  within  SOAC  Ascendancy 

17.5% 

5.6% 

14.8% 

%  of  Total 

13.6% 

1.3% 

14.8% 

-1.08  SD 

Below  Avg. 

Count 

347 

57 

404 

%  within  SORT  Category 

85.9% 

14.1% 

100.0% 

%  within  SOAC  Ascendancy 

31.1% 

17.6% 

28.1% 

%  of  Total 

24.1% 

4.0% 

28.1% 

-.23  SD 

Avg. 

Count 

431 

155 

586 

%  within  SORT  Category 

73.5% 

26.5% 

100.0% 

%  within  SOAC  Ascendancy 

38.7% 

48.0% 

40.8% 

%  of  Total 

30.0% 

10.8% 

40.8% 

-1-1.02  SD 

Above  Avg. 

Count 

105 

57 

162 

%  within  SORT  Category 

64.8% 

35.2% 

100.0% 

%  within  SOAC  Ascendancy 

9.4% 

17.6% 

11.3% 

%  of  Total 

7.3% 

4.0% 

11.3% 

-1-1.78  SD 

High 

Count 

36 

36 

72 

%  within  SORT  Category 

50.0% 

50.0% 

100.0% 

%  within  SOAC  Ascendancy 

3.2% 

11.1% 

5.0% 

%  of  Total 

2.5% 

2.5% 

5.0% 

Total 

Count 

1114 

323 

1437 

%  within  SORT  Category 

77.5% 

22.5% 

100.0% 

%  within  SOAC  Ascendancy 

100.0% 

100.0% 

100.0% 

%  of  Total 

77.5% 

22.5% 

100.0% 

As  ean  be  seen  from  Table  5,  the  SORT  eategories  were  labeled  as  low,  below  average,  average, 
above  average,  and  high.  These  SORT  category  labels  correspond  to  the  probability  of  advancing 
to  SOAC  when  taking  the  base  rate  of  ascendancy  into  account,  which  is  highlighted  in  green 
and  approximately  23%  aeeording  to  these  results.  For  example,  SOBC  students  who  fell  into  the 
high  eategory  on  the  SORT  (a  total  of  72  out  of  1437  students)  had  a  50%  probability  of 
advancing  to  SOAC  (highlighted  in  yellow)  in  comparison  to  the  base  rate  ascendancy  of  23%. 
Note  that  for  SOBC  students  to  be  placed  into  the  high  category  they  had  to  obtain  SORT  scores 
that  were  at  least  +1.78  standard  deviations  {SD)  above  the  mean,  or  higher.  The  other  standard 
deviations  to  whieh  outpoints  for  the  eategories  on  the  SORT  oorrespond  ean  be  seen  in  the  far- 
left  column  of  the  table  above  the  +1.78  SD  value,  which  is  adjacent  to  the  high  SORT  category. 
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Thus,  for  example,  those  in  the  low  SORT  category  were  -1.08  SD  beneath  the  mean  or  lower  on 
the  SORT. 


In  sum,  we  were  able  to  identify  various  outpoints  on  the  SORT  that  could  be  used  to  enhance 
retention  or  selection  decisions  for  submarine  training.  Notice  that  were  SUBFOR  to  use  the 
SORT  by  eliminating  from  the  submarine  training  pipeline  all  those  who  fall  into  the  low  SORT 
category  (i.e.,  the  bottom  14.8%)  because  they  are  not  likely  to  ascend  to  SOAC,  then  SUBFOR 
would  be  incorrect  only  18  out  of  213  times,  or  8.5%  of  the  time.  Likewise,  were  SUBFOR  to 
use  SORT  by  eliminating  all  those  who  fall  into  the  below  average  category  or  lower  (i.e.,  the 
bottom  42.9%)  because  they  are  not  likely  to  ascend  to  SOAC,  then  SUBFOR  would  be  incorrect 
only  75  out  of  617  times,  or  12.2%  of  the  time. 

DISCUSSION 

The  current  study  clearly  indicates  that  SUBSCREEN  test  scores  can  predict  ascendancy  from 
SOBC  to  SOAC.  The  motivational  scales  of  which  SUBSCREEN’s  SORT  is  composed  were 
hypothesized  to  be  the  best  predictors  of  ascendancy,  and  this  hypothesis  was  confirmed.  As 
increases  in  motivation  are  known  to  lead  to  increases  in  job  performance  (Barrick,  Stewart,  & 
Piotrowski,  2002;  Helmreich,  Sawin,  &  Carsrud,  1986;  Kirk  &  Brown,  2003;  Mitchell,  1997), 
and  only  high  performing  officers  are  selected  to  attend  SOAC,  perhaps  this  result  is  not 
surprising. 

However,  the  fact  that  SUBSCREEN’s  SORT  predicts  ascendancy  from  SOBC  to  SOAC  with  R 
=  .250  (p  <  .001)  could  also  be  viewed  as  both  potentially  useful  and  perhaps  profound  for  the 
following  reasons.  Eirst,  there  is  extensive  range  restriction  among  the  current  study  participants, 
and  range  restriction  decreases  the  predictive  power  of  screening  instruments,  psychological 
tests,  and  measures  of  individual  differences  in  general  (Cohen  &  Cohen,  1983;  Hunter  & 
Schmidt,  1990;  Nunnally  &  Bernstein,  1994;  Pearson,  1903;  Powers,  2004).  To  understand  the 
extent  of  the  range  restriction  in  the  sample  investigated,  it  should  be  noted  that  administration  of 
the  SUBSCREEN  test  occurred  at  SOBC  for  these  study  participants.  In  order  to  enter  SOBC,  a 
submarine  officer  must  have  graduated  from  college  with  very  respectable  grades,  and  many 
attend  the  Naval  Academy,  which  is  a  very  selective  institute.  Eurthermore,  these  college 
graduates  must  all  pass  security  clearance  procedures.  Additionally,  they  must  complete 
accession.  Nuclear  Power  School,  and  Nuclear  Prototype  School  in  order  to  enter  SOBC.  This 
training  is  generally  equivalent  to  obtaining  a  Masters  in  Nuclear  Engineering,  and  is 
exceedingly  rigorous.  Thus,  as  those  with  less  motivation  and  ability  attrite  along  this  career  path 
at  various  points  (e.g.,  during  college,  during  Nuclear  Power  School,  etc.),  it  is  reasonable  to 
assert  that  any  sample  of  SOBC  students  is  greatly  restricted  in  range  with  respect  to  many 
characteristics  as  these  students  belong  to  a  highly  motivated  and  highly  intelligent  population. 

In  sum,  the  current  findings  indicate  that  SUBSCREEN’s  SORT  was  capable  of  capturing 
individual  differences  in  motivational  traits  among  SOBC  students  that  predicted  later 
advancement  to  SOAC,  and  we  can  conclude  that  this  result  occurred  in  spite  of  any  range 
restriction  in  the  sample.^ 


^  Appendix  E  provides  a  mathematical  estimation  of  the  amount  of  range  restriction  in  the  SOBC  sample  and  a 
subsequent  re -estimation  of  the  relationship  between  the  SORT  and  SOAC  ascendancy  after  correcting  for  range 
restriction  in  the  predictor  (i.e.,  the  SORT). 
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Second,  the  time  span  between  administration  of  the  SUBSCREEN  test  at  SOBC  and  later 
advancement  to  SOAC  is  approximately  seven  years.  Consequently,  advancement  to  SOAC  is  a 
very  temporally  distal  outcome  in  light  of  the  acquisition  of  the  motivational  attributes  with  the 
SUBSCREEN  test  at  SOBC,  and  yet  SUBSCREEN’ s  SORT  still  predicted  SOAC  ascendancy. 
The  fact  that  among  enlisted  personnel  SUBSCREEN’ s  SMART  predicts  indicators  of  training 
attrition  acquired  months  after  test  administration  (e.g.,  correlations  of  .35  and  .50,  p  <  .001) 
somewhat  better  than  fleet  attrition  acquired  several  years  later  (e.g.,  .25,  p  <  .001;  Bing  & 
Eisenberg,  2004)  does  suggest  that  perhaps  changes  in  persons  over  time  and  intervening 
situational  events  can  attenuate  the  predictive  power  of  the  test,  but  over  the  span  of  many  years. 
Thus,  once  again,  in  spite  o/the  large  time  span  (i.e.,  approximately  seven  years)  between  test 
administration  and  the  outcome  (i.e.,  SOAC  ascendancy)  studied  here,  the  SORT  was  still 
capable  of  capturing  stable  motivational  attributes  that  were  manifested  in  terms  of  fleet  attrition 
for  some,  and  high  job  performance  and  advancement  to  SOAC  for  others. 

Third,  the  base  rate  of  the  outcome,  SOAC  ascendancy,  was  approximately  23%  in  the  sample 
studied  here.  This  low  base  rate  also  attenuates  the  relation  (i.e.,  correlation  or  R)  observed 
between  SUBSCREEN’ s  SORT  and  SOAC  ascendancy.  This  attenuation  in  observed  relations 
will  always  occur  when  the  predictor  is  a  continuous  score,  like  the  SORT,  the  outcome  is 
dichotomous,  and  the  base  rate  of  occurrence  in  the  dichotomous  outcome  deviates  from  a  50/50 
(present/absent)  split  (Cohen  &  Cohen,  1983;  Nunnally  &  Bernstein,  1994).  In  the  current  data, 
with  a  23/77  split  (i.e.,  a  23%  base  rate)  on  the  SOAC  ascendancy  outcome,  the  maximum 
possible  correlation  or  R  between  the  SORT  and  SOAC  ascendancy  was  approximately  .730 
rather  than  1.00.  As  a  consequence,  once  again,  the  observed  R  of  .250  ip  <  .001)  between  the 
SORT  and  the  SOAC  ascendancy  outcome  was  observed  in  spite  o/this  deviation  from  a  50/50 
split  in  the  dichotomous  SOAC  outcome.  Were  we  to  also  assume  that  the  dichotomous  SOAC 
ascendancy  outcome  was  a  manifestation  of  a  continuous  and  normally  distributed  construct 
(e.g.,  job  performance,  leadership  ability),  then  the  observed  R  could  be  re-estimated.  In  fact, 
when  assuming  the  dichotomous  SOAC  outcome  is  a  function  of  a  continuous,  normally 
distributed  construct,  re-estimating  the  current  R  of  .250,  which  is  essentially  a  point  biserial 
correlation  between  SORT  and  the  dichotomous  SOAC  outcome,  results  in  a  biserial  correlation 
of  .347  (Cohen  &  Cohen,  1983). 

East,  we  could  also  interpret  the  .250  relation  observed  here  for  the  SORT  in  the  context  of  well- 
conducted  meta-analyses  investigating  the  relationship  between  personality  measures  of 
Conscientiousness  and  job  performance.  Among  personality  traits,  the  Big  Eive  personality 
factor  of  Conscientiousness  has  been  recognized  as  the  strongest  and  most  generalizable 
predictor  of  job  performance  (Barrick  &  Mount,  1991;  Hurtz  &  Donovan,  2000;  Tett,  Jackson,  & 
Rothstein,  1991).^  However,  various  meta-analyses  have  demonstrated  that  the  uncorrected  mean 
validity  coefficients  between  self-reports  of  Conscientiousness  and  measures  of  job  performance 
range  from  .12  to  .18  (Barrick  &  Mount,  1991;  Hough,  Eaton,  Dunnette,  Kamp,  &  McCloy, 

1990;  Hurtz  &  Donovan,  2000;  Mount  &  Barrick,  1995;  Tett  et  ah,  1991).  Thus,  in  the  context  of 


®  The  Big  Five  taxonomy  characterizes  personality  as  being  composed  of  five  major  second  order  factors: 
Conscientiousness,  Extraversion,  Openness,  Agreeableness,  and  Emotional  Stability.  Of  these  factors. 
Conscientiousness  is  generally  recognized  as  the  most  work-relevant  for  the  majority  of  occupations. 
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these  meta-analytic  findings,  the  .250  observed  here  for  the  SORT  in  predicting  SOAC 
ascendancy  is  a  respectable  value. 

In  sum,  we  may  reasonably  assume  that  the  relationship  found  here  between  SUBSCREEN’ s 
SORT  and  advancement  to  SOAC  is  a  lower  bound  estimate  of  the  true  relationship,  and  were 
the  SORT  to  be  implemented  for  submarine  training  selection  decisions  then  the  Navy  may 
accrue  several  financial  benefits.  These  financial  benefits  are  subsequently  discussed  among 
other  conclusions  and  recommendations. 

CONCLUSIONS  AND  RECOMMENDATIONS 

The  current  findings  indicate  that  SUBSCREEN’ s  SORT  captures  stable  individual  differences 
in  motivation  that  influence  a  submariner’s  ability  to  ultimately  ascend  to  higher  leadership 
positions  within  SUBEOR.  The  outpoint  analyses  pursued  here  also  indicated  that  the  SORT  may 
provide  an  actionable  metric  that  can  improve  submarine  training  selection  decisions,  especially 
if  used  to  eliminate  the  low  and  below  average  SORT  categories,  which  had  only  a  12.2%  chance 
of  ascending  to  SOAC.  This  would  be  a  large  force  reduction,  eliminating  approximately  43%  of 
the  candidate  pool,  but  this  portion  of  the  candidate  pool  has  a  12.2%  chance  of  ascending  to 
SOAC  in  comparison  to  the  base  rate  of  23%  (i.e.,  about  half  the  chance  of  the  randomly  selected 
candidate).  This  type  of  use  of  the  SORT  would  ultimately  help  the  Navy  to  meet  HCS 
objectives,  which  include  attempts  to  use  actionable  metrics  to  decrease  overall  manpower  costs 
while  maintaining  operational  effectiveness.  Additionally,  the  use  of  the  SORT  to  eliminate 
officers  from  the  submarine  training  pipeline  need  not  lead  to  a  lack  of  career  advancement  for 
all  of  these  officers  as  they  (1)  would  be  removed  from  a  career  path  in  SUBEOR  where  they  are 
not  predicted  to  be  successful  and  (2)  could  be  placed  into  more  suitable  career  paths  where 
retention  and  advancement  are  more  likely. 

In  order  to  avoid  spending  training  dollars,  in  particular  the  large  sums  of  money  needed  for 
nuclear  power  training,  on  officers  in  SUBEOR  who  are  not  likely  to  ascend  to  SOAC,  the  Navy 
could  consider  moving  the  administration  of  the  SUB  SCREEN  test  to  earlier  stages  in  the 
submarine  training  pipeline.  As  force  reduction  initiatives  increase,  administration  of  the 
SUBSCREEN  test  prior  to  Nuclear  Power  School  and  the  use  of  the  SORT  for  selection 
decisions  could  save  the  Navy  a  substantial  amount  of  money  by  reducing  the  overall  number  of 
submarine  officer  trainees  while  simultaneously  retaining  the  trainees  that  are  the  most  likely  to 
advance  in  their  submarine  careers. 

There  are  two  caveats  to  this  recommendation.  Eirst,  as  SUBSCREEN  is  currently  administered 
during  SOBC,  and  SOBC  students  have  completed  an  extensive  amount  of  nuclear  power 
training  by  the  time  they  take  the  test,  the  attitude  toward  item  responses  by  the  average  SOBC 
student  may  be  very  different  than  the  attitude  of  a  pre-Nuclear  Power  School  submarine  officer 
candidate.  This  possible  change  in  test-taker  attitude  and  motivation  could  attenuate  the  validity 
of  the  test,  and  thus  the  predictive  power  of  SUBSCREEN’ s  SORT  (Bing,  Whanger,  Davison,  & 
VanHook,  2004).  However,  several  scales  of  which  the  SORT  is  composed  are  context- specific, 
and  contextualized  to  the  submarine  service.  Research  in  personality  assessment  has  shown  that 
scales  with  a  high  degree  of  context- specificity  may  maintain  their  validity  even  in  the  presence 
of  situational  factors  that  cause  respondents  to  engage  in  response  distortion  via  socially 
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desirable  responding  or  faking  good  attempts  (Bing  et  al.,  2004;  Schmit,  Ryan,  Stierwalt,  & 
Powell,  1995).  Also,  SORT’S  RSI  scale  is  sensitive  to  response  distortion,  and  RSI  scale  scores 
increase  as  response  distortion  increases.  Thus,  because  RSI  is  a  component  of  the  SORT,  and 
negatively  weighted  in  the  SORT  equation,  SORT  scores  will  be  resilient  to  faking  attempts. 
Specifically,  if  faking  attempts  increase  among  respondents,  then  SORT  scores  will  subsequently 
decrease  for  these  respondents,  leading  to  the  prediction  that  they  are  not  likely  to  ascend  in 
SUBFOR  according  to  their  SORT  scores. 

Second,  if  the  administration  of  the  SUBSCREEN  test  were  to  be  moved  to  an  earlier  point  in  the 
training  pipeline,  then  the  mental  health  examination  referral  process  used  to  meet  the 
MANMED  requirement  for  submarine  duty  articulated  in  Article  15-69,  paragraph  2(i)  of  that 
document  (see  Appendix  B)  may  need  to  be  transferred  to  that  same  point  as  well,  and  thus  to  the 
same  location  as  the  administration  site  of  the  SUBSCREEN  test.  However,  because  the 
MANMED  also  mandates  psychological  screening  for  nuclear  field  duty  in  Article  15-70, 
paragraph  2(c)  of  that  document,  perhaps  the  psychological  screenings  for  both  duties  could 
occur  at  the  same  point  in  the  training  pipeline,  which  would  eliminate  some  redundancy  of 
operation  and  possibly  lead  to  fiscal  savings  for  the  Navy.^ 

In  sum,  moving  the  administration  of  the  SUB  SCREEN  test  upstream  in  the  training  pipeline 
could  result  in  substantial  financial  savings  for  the  Navy,  and  the  SORT’S  validity  should  not  be 
overly  attenuated  as  a  consequence.  Also,  as  less  range  restriction  would  be  present  among  pre- 
Nuclear  Power  School  submarine  officer  candidates,  the  SORT’S  validity  may  ultimately  remain 
stable  under  this  potential  administrative  change  in  testing  conditions.  Additional  investigations 
into  the  validity  of  the  SUBSCREEN  test  and  the  SORT  should  be  undertaken  if  and  when  the 
administration  of  the  SUB  SCREEN  test  is  moved  to  an  earlier  stage  in  the  submarine  training 
pipeline  to  meet  HCS  objectives. 


^  MANMED  Article  15-70,  Nuclear  Field  Duty  (Nuclear  Power  /  Nuclear  Weapons),  paragraph  2(c)  begins  as 
follows:  “Psychiatric .  Because  of  the  potential  for  misuse  of  devices  and  sources  emitting  ionizing  radiation,  the 
psychological  fitness  of  applicants  must  be  carefully  appraised  by  the  examining  physician.  The  objective  is  to  elicit 
evidence  of  tendencies  which  militate  against  assignment  to  these  critical  duties.”  It  should  be  noted  that  seven  of 
the  eight  tendencies  listed  in  this  Article  which  prevent  assignment  to  nuclear  field  duty  are  also  mentioned  in 
Article  15-69,  paragraph  2(i),  and  also  prevent  assignment  to  submarine  duty.  Thus,  the  redundancy  in  the  traits  or 
tendencies  that  prevent  assignment  to  these  duties  could  potentially  be  assessed  at  one  point  in  the  training  pipeline. 
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APPENDIX  A.  Naval  Submarine  School  (NAVSUBSCOL)  Instruction  6420.1,  Excerpt 


Subj :  NAVAL  SUBMARINE  SCHOOL  SUBSCREEN  ADMINISTRATION 

Ref:  (a)  Manual  of  the  Medical  Department  (MANMED  PI  17) 

Enel:  (1)  Basic  Enlisted  Submarine  School  (BESS)  SUBSCREEN 

Administration  Procedures 

1 .  Purpose.  To  establish  formal  procedures  for  administering  SUBSCREEN  to  submarine 
force  officer  and  enlisted  accessions  and  for  documenting  the  SUBSCREEN  process  at  Naval 
Submarine  School. 

2.  Discussion.  Reference  (a)  states  that  the  psychological  fitness  of  applicants  for 
submarine  training  must  be  carefully  appraised.  SUBSCREEN  was  developed  for  this  purpose 
and  is  administered  to  all  submarine  officer  and  enlisted  basic  course  students  at  Naval 
Submarine  School. 

3.  Action 


a.  Director,  General  Skills  Training  (N62)  shall  institute  the  procedure  delineated  in 
enclosure  (1)  for  BESS  students. 

b.  Director,  Submarine  Officer  Basic  Course  (SOBC)  (N22)  shall  institute  the  procedures 
delineated  in  enclosure  (2)  for  SOBC  students. 

c.  Directors  of  SOBC  (N22)  and  General  Skills  Training  (N62)  are  jointly  responsible 
for  updating  this  instruction  when  necessary. 

W.  A.  Peters  (signed) 


Distribution: 

CO/XO 

Admin 

N2 

N6 

Mental  Health 
NSMRL  (3) 

LT  Davis  (Sick  Call) 

4.  Scoring  and  Evaluation 

a.  The  NSMRL  Examiner  scores  SUBSCREEN.  The  evaluation  criteria  are  then  applied 
to  SUBSCREEN  scale  scores.  The  NSMRL  Examiner  records  on  the  class  roster  whether  or  not 
a  student’s  score  warrants  referral  to  the  Mental  Health  Clinic  (MHC). 
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APPENDIX  B.  Manual  of  the  Medical  Department  (MANMED)  Article  15-69  para  2(i) 


''Psychiatric.  Because  of  the  nature  of  the  duties  and  responsibilities  of  each  person  in  a 
submarine,  the  psychological  fitness  of  applicants  for  submarine  training  must  be  carefully 
appraised.  The  objective  is  to  elicit  evidence  of  tendencies  which  might  prevent  satisfactory 
adjustment  to  submarine  life.  Among  these  are  below  average  intelligence,  claustrophobic 
tendencies,  lack  of  motivation,  unhealthy  motivation,  history  of  personal  ineffectiveness, 
difficulties  in  interpersonal  relations,  lack  of  adaptability,  or  personality  disorders. 

(1)  Any  examinee  diagnosed  by  a  psychiatrist,  clinical  psychologist,  or  UMO  as  suffering 
from  depression,  psychosis,  manic-depression,  paranoia,  severe  neurosis,  severe  borderline 
personality,  or  schizophrenia  will  be  recommended  for  submarine  disqualification  at  the  time  of 
initial  diagnosis.  Waiver  request  may  be  submitted  per  section  V. 

(2)  Those  personnel  with  diagnosed  suicidal  ideation  will  have  their  cases  reviewed,  as  a 
minimum,  by  the  type  commander  (TYCOM)  medical  officer,  if  a  UMO,  for  fleet  personnel,  or 
MED-21  if  at  a  shore  establishment,  to  determine  the  necessity  for  disqualification  or  return  to 
duty.  Personnel  with  suicidal  gestures  or  attempts  will  be  recommended  for  submarine 
disqualification.  Waivers  will  be  considered  on  individual  basis  per  section  V. 

(3)  Those  personnel  with  minor  psychiatric  disorders  such  as  acute  situational  stress  reactions 
will  be  evaluated  by  the  local  group  or  squadron  UMO  in  conjunction  with  a  formal  psychiatric 
evaluation  when  necessary.  Those  cases  which  resolve  completely,  quickly,  and  without 
significant  psychotherapy  can  be  found  fit  for  submarine  duty  by  the  responsible  local  UMO,  if 
deemed  appropriate.  Those  cases  in  which  confusion  exists  must  be  reviewed  by  the  TYCOM 
medical  officer,  if  a  UMO,  for  fleet  personnel,  or  MED-21  for  shore-based  personnel.  It  must  be 
stressed  that  any  consideration  for  return  to  duty  in  these  cases  must  address  the  issue  of  whether 
to  service  member,  in  the  written  opinions  of  the  UMO  and  the  member’s  commanding  officer, 
can  successfully  return  to  the  specific  stresses  and  environment  of  submarine  duty.” 
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APPENDIX  C.  Implications  of  the  Uniform  Guidelines  on  Employee  Selection  Procedures 
(1978)  for  the  Use  of  SUBSCREEN’s  SMART  and  the  SORT 


With  respect  to  the  current  6.4,  BUMED-funded  Test  and  Evaluation  (T&E)  of  the  SMART,  it  is 
relevant  that  the  Eederal  Government,  via  the  Uniform  Guidelines  on  Employee  Selection 
Procedures  (1978),  does  not  make  a  distinction  between  tests  like  SUBSCREEN,  which  are  self- 
report  questionnaires  of  attitudes,  personality,  motivation,  environmental  adaptability  and 
psychological  functioning,  and  tests  like  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB),  which  is  a  cognitive  ability  test.  As  such,  it  is  common  practice  for  those  with  a  need 
to  know  to  receive  scores  on  tests  to  make  informed  decisions  regarding  not  only  whether  or  not 
employees  in  both  the  private  sector  and  the  government  (e.g.,  military  personnel)  should  be 
trained,  retrained,  demoted,  or  referred,  etc.,  as  specified  in  the  Uniform  Guidelines  with  respect 
to  the  scope  of  test  use  (see  below),  but  also  whether  or  not  the  test  is  working  to  improve  these 
decisions.  Thus,  for  example,  many  instructors,  chiefs,  and  other  leaders  at  NAVSUBSCOE  have 
access  to  ASVAB  scores  of  BESS  students  because  they  need  those  scores  to  make  decisions 
about  their  training,  transfer,  rate  qualification,  etc. 

Thus,  we  have  good  reason  to  believe  that  NAVSUBSCOE  will  use  the  SMART  results 
appropriately  given  their  extensive  experience  not  only  with  testing  data,  such  as  class  grades, 
the  ASVAB,  and  the  current  “referred  versus  not  referred”  SUBSCREEN  information  that  they 
obtain  from  the  SUB  SCREEN  program,  but  also  because  they  are  very  experienced  in  routinely 
dealing  with  sensitive  information  on  their  students,  such  as  security  clearance  information. 

It  should  be  noted  that  the  Uniform  Guidelines  on  Employee  Selection  Procedures  (1978)  apply 
to  more  than  just  selection  decisions.  Prom  Section  2  of  the  Uniform  Guidelines  under  “B. 
Employment  decisions”  we  have  the  following  quote  from  the  guidelines: 

“These  guidelines  apply  to  tests  and  other  selection  procedures  which  are  used  as 
a  basis  for  any  employment  decision.  Employment  decisions  include  but  are  not 
limited  to  hiring,  promotion,  demotion,  membership  (for  example,  in  a  labor 
organization),  referral,  retention,  and  licensing  and  certification,  to  the  extent 
that  licensing  and  certification  may  be  covered  by  Eederal  equal  employment 
opportunity  law.  Other  selection  decisions,  such  as  selection  for  training  or 
transfer,  may  also  he  considered  employment  decisions  if  they  lead  to  any  of 
the  decisions  listed  above. 

As  such,  it  can  be  concluded  that  a  legal  precedent  is  available  for  the  release  of  the  SMART  to 
NAVSUBSCOE,  and  for  NAVSUBSCOE’ s  use  of  the  SMART  in  the  psychological  screening  of 
future  prospective  submariners.  Purthermore,  as  the  SORT  is  a  subset  of  the  SUBSCREEN  test, 
this  legal  precedent  applies  to  the  use  of  SORT  for  employment  decisions  as  well. 


*  Bold  font  is  added  for  emphasis  and  does  not  appear  in  the  original  text. 
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APPENDIX  D.  Names  of  SUBSCREEN’s  SORT  Scales  and  the  Respective  Items 


Appendix  D  is  for  Official  Use  Only,  and  any  request  for  disclosure  of  any  information 
contained  in  this  appendix  should  be  forwarded  to  the  Commanding  Officer,  Naval  Submarine 
Medical  Research  Laboratory,  Groton,  CT. 
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APPENDIX  E.  Re-estimation  of  the  Relationship  Between  the  SORT  and  SOAC 
Ascendancy  after  Correcting  for  Predictor  Range  Restriction 

In  order  to  estimate  the  amount  of  range  restriction  on  SORT  scores  for  SOBC  students  (n  = 
1,437),  we  calculated  SORT  scores  for  a  large  sample  of  BESS  students  (n  =  25,277)  from  their 
SUBSCREEN  test  results.  The  majority  of  the  BESS  student  population  is  composed  of  high 
school  graduates  who  enter  BESS  after  completing  basic  training.  However,  it  should  be  noted 
that  (1)  SORT  scores  are  largely  a  function  of  motivational  traits  (see  Table  1),  (2)  BESS 
students  are  highly  motivated  to  enter  the  Submarine  Eorce,  which  is  why  they  volunteered  for 
submarine  duty,  and  (3)  there  are  minimum  ability  requirements  to  enter  BESS,  which  restricts 
the  range  of  BESS  students  on  ability  attributes  as  well.  Eollowing  Cohen  and  Cohen  (1983),  we 
used  the  standard  deviation  {SD)  on  the  SORT  scores  from  the  BESS  student  sample  to  re- 
estimate  the  relationship  between  the  SORT  and  SOAC  ascendancy  among  SOBC  students. 
Eirst,  note  that  the  SD  on  the  SORT  for  SOBC  students  was  approximately  .100  (i.e.,  .09962), 
whereas  it  was  approximately  .112  (i.e.,  .11175)  for  BESS  students.  Thus,  SORT  scores  in  the 
SOBC  student  sample  were  restricted  in  range  by  approximately  1 1%  in  comparison  to  the 
dispersion  of  SORT  scores  found  in  the  BESS  student  sample.  Using  Cohen  and  Cohen’s  (1983, 
p.  70)  formula,  we  found  that  when  correcting  for  this  estimate  of  range  restriction  in  SORT 
scores  among  SOBC  students  the  relationship  between  SORT  and  SOAC  ascendancy  increased 
from  the  observed  R  of  .250  to  .280. 
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